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Abstract 

Understanding the ways in which information 
achieves widespread public awareness is a re- 
search question of significant interest. We 
consider whether, and how, the way in which 
the information is phrased — the choice of 
words and sentence structure — can affect this 
process. To this end, we develop an analy- 
sis framework and build a corpus of movie 
quotes, annotated with memorability informa- 
tion, in which we are able to control for both 
the speaker and the setting of the quotes. We 
find that there are significant differences be- 
tween memorable and non-memorable quotes 
in several key dimensions, even after control- 
ling for situational and contextual factors. One 
is lexical distinctiveness: in aggregate, memo- 
rable quotes use less common word choices, 
but at the same time are built upon a scaf- 
folding of common syntactic patterns. An- 
other is that memorable quotes tend to be more 
general in ways that make them easy to ap- 
ply in new contexts — that is, more portable. 
We also show how the concept of "memorable 
language" can be extended across domains. 

To appear at ACL 2012; final version 

1 Hello. My name is Inigo Montoya. 

Understanding what items will be retained in the 
public consciousness, and why, is a question of fun- 
damental interest in many domains, including mar- 
keting, politics, entertaiimient, and social media; as 
we all know, many items barely register, whereas 
others catch on and take hold in many people's 
minds. 



An active line of recent computational work has 
employed a variety of perspectives on this question. 
Building on a foundation in the sociology of diffu- 
sion EtIIBTI . researchers have explored the ways in 
which network structure affects the way information 
spreads, with domains of interest including blogs 
fflHIl, email ifJTl . on-line commerce 1*221, and so- 
cial media El|28l|33l|38l- There has also been recent 
research addressing temporal aspects of how differ- 
ent media sources convey information |[23l l30l |39l 
and ways in which people react differently to infor- 
mation on different topics ll28l [36l . 

Beyond all these factors, however, one's everyday 
experience with these domains suggests that the way 
in which a piece of information is expressed — the 
choice of words, the way it is phrased — might also 
have a fundamental effect on the extent to which it 
takes hold in people's minds. Concepts that attain 
wide reach are often carried in messages such as 
political slogans, marketing phrases, or aphorisms 
whose language seems intuitively to be memorable, 
"catchy," or otherwise compelling. 

Our first challenge in exploring this hypothesis is 
to develop a notion of "successful" language that is 
precise enough to allow for quantitative evaluation. 
We also face the challenge of devising an evaluation 
setting that separates the phrasing of a message from 
the conditions in which it was delivered — highly- 
cited quotes tend to have been delivered under com- 
pelling circumstances or fit an existing cultural, po- 
litical, or social narrative, and potentially what ap- 
peals to us about the quote is really just its invoca- 
tion of these extra-linguistic contexts. Is the form 
of the language adding an effect beyond or indepen- 



dent of these (obviously very crucial) factors? To 
investigate the question, one needs a way of control- 
ling — as much as possible — for the role that the 
surrounding context of the language plays. 

The present work (i): Evaluating language-based 
memorability Defining what makes an utterance 
memorable is subtle, and scholars in several do- 
mains have written about this question. There is 
a rough consensus that an appropriate definition 
involves elements of both recognition — people 
should be able to retain the quote and recognize it 
when they hear it invoked — &nA production — peo- 
ple should be motivated to refer to it in relevant sit- 
uations iflSl . One suggested reason for why some 
memes succeed is their ability to provoke emotions 
|[T6l . Alternatively, memorable quotes can be good 
for expressing the feelings, mood, or situation of an 
individual, a group, or a culture (the Zeitgeist): "Cer- 
tain quotes exquisitely capture the mood or feeling 
we wish to communicate to someone. We hear them 
... and store them away for future use" [10|. 

None of these observations, however, serve as 
definitions, and indeed, we believe it desirable to 
not pre-commit to an abstract definition, but rather 
to adopt an operational formulation based on exter- 
nal human judgments. In designing our study, we 
focus on a domain in which (i) there is rich use of 
language, some of which has achieved deep cultural 
penetration; (ii) there already exist a large number of 
external human judgments — perhaps implicit, but 
in a form we can extract; and (iii) we can control for 
the setting in which the text was used. 

Specifically, we use the complete scripts of 
roughly 1000 movies, representing diverse genres, 
eras, and levels of popularity, and consider which 
lines are the most "memorable". To acquire memo- 
rability labels, for each sentence in each script, we 
determine whether it has been listed as a "memo- 
rable quote" by users of the widely-known IMDb 
(the Internet Movie Database), and also estimate the 
number of times it appears on the Web. Both of these 
serve as memorability metrics for our purposes. 

When we evaluate properties of memorable 
quotes, we compare them with quotes that are not as- 
sessed as memorable, but were spoken by the same 
character, at approximately the same point in the 
same movie. This enables us to control in a fairly 



fine-grained way for the confounding effects of con- 
text discussed above: we can observe differences 
that persist even after taking into account both the 
speaker and the setting. 

In a pilot validation study, we find that human 
subjects are effective at recognizing the more IMDb- 
memorable of two quotes, even for movies they have 
not seen. This motivates a search for features in- 
trinsic to the text of quotes that signal memorabil- 
ity. In fact, comments provided by the human sub- 
jects as part of the task suggested two basic forms 
that such textual signals could take: subjects felt that 
(i) memorable quotes often involve a distinctive turn 
of phrase; and (ii) memorable quotes tend to invoke 
general themes that aren't tied to the specific setting 
they came from, and hence can be more easily in- 
voked for future (out of context) uses. We test both 
of these principles in our analysis of the data. 

The present work (ii): What distinguishes mem- 
orable quotes Under the controlled-comparison 
setting sketched above, we find that memorable 
quotes exhibit significant differences from non- 
memorable quotes in several fundamental respects, 
and these differences in the data reinforce the two 
main principles from the human pilot study. First, 
we show a concrete sense in which memorable 
quotes are indeed distinctive: with respect to lexi- 
cal language models trained on the newswire por- 
tions of the Brown corpus flT], memorable quotes 
have significantly lower likelihood than their non- 
memorable counterparts. Interestingly, this distinc- 
tiveness takes place at the level of words, but not 
at the level of other syntactic features: the part-of- 
speech composition of memorable quotes is in fact 
more likely with respect to newswire. Thus, we can 
think of memorable quotes as consisting, in an ag- 
gregate sense, of unusual word choices built on a 
scaffolding of common part-of-speech patterns. 

We also identify a number of ways in which mem- 
orable quotes convey greater generality. In their pat- 
terns of verb tenses, personal pronouns, and deter- 
miners, memorable quotes are structured so as to be 
more "free-standing," containing fewer markers that 
indicate references to nearby text. 

Memorable quotes differ in other interesting as- 
pects as well, such as sound distributions. 

Our analysis of memorable movie quotes suggests 



a framework by which the memorabiUty of text in 
a range of different domains could be investigated. 
We provide evidence that such cross-domain prop- 
erties may hold, guided by one of our motivating 
applications in marketing. In particular, we analyze 
a corpus of advertising slogans, and we show that 
these slogans have significantly greater likelihood 
at both the word level and the part-of-speech level 
with respect to a language model trained on mem- 
orable movie quotes, compared to a corresponding 
language model trained on non-memorable movie 
quotes. This suggests that some of the principles un- 
derlying memorable text have the potential to apply 
across different areas. 

Roadmap ^lays the empirical foundations of our 
work: the design and creation of our movie-quotes 



dataset, which we make publicly available (^2.1 1, a 
pilot study with human subjects validating IMDb- 
based memorability labels ( |2.2[ ), and further study 
of incorporating search-engine counts {{ 2.3 1, ^de- 
tails our analysis and prediction experiments, using 
both movie-quotes data and, as an exploration of 
cross-domain applicability, slogans data, ^surveys 
related work across a variety of fields. ^ briefly 
summarizes and indicates some future directions. 

2 I'm ready for my close-up. 

2.1 Data 

To study the properties of memorable movie quotes, 
we need a source of movie lines and a designation 
of memorability. Following [8|, we constructed a 
corpus consisting of all lines from roughly 1000 
movies, varying in genre, era, and popularity; for 
each movie, we then extracted the list of quotes from 
IMDb's Memorable Quotes page corresponding to 
the movie{3 

A memorable quote in IMDb can appear either as 
an individual sentence spoken by one character, or 
as a multi-sentence line, or as a block of dialogue in- 
volving multiple characters. In the latter two cases, 
it can be hard to determine which particular portion 
is viewed as memorable (some involve a build-up to 
a punch line; others involve the follow-through after 
a well-phrased opening sentence), and so we focus 




Figure 1 : Location of memorable quotes in each decile 
of movie scripts (the first 10th, the second 10th, etc.), 
summed over all movies. The same qualitative results 
hold if we discard each movie's very first and last line, 
which might have privileged status. 

in our comparisons on those memorable quotes that 
appear as a single sentence rather than a multi-line 
blockE] 

We now formulate a task that we can use to eval- 
uate the features of memorable quotes. Recall that 
our goal is to identify effects based in the language 
of the quotes themselves, beyond any factors arising 
from the speaker or context. Thus, for each (single- 
sentence) memorable quote M, we identify a non- 
memorable quote that is as similar as possible to M 
in all characteristics but the choice of words. This 
means we want it to be spoken by the same chai^ac- 
ter in the same movie. It also means that we want 
it to have the same length: controlling for length is 
important because we expect that on average, shorter 
quotes will be easier to remember than long quotes, 
and that wouldn't be an interesting textual effect to 
report. Moreover, we also want to control for the 
fact that a quote's position in a movie can affect 
memorability: certain scenes produce more mem- 
orable dialogue, and as Figure [T] demonstrates, in 
aggregate memorable quotes also occur dispropor- 
tionately near the beginnings and especially the ends 
of movies. In summary, then, for each M, we pick a 
contrasting (single-sentence) quote N from the same 
movie that is as close in the script as possible to M 
(either before or after it), subject to the conditions 
that (i) M and N are uttered by the same speaker, 
(ii) M and N have the same number of words, and 



This extraction involved some edit-distance-based align- 
ment, since the exact form of the line in the script can exhibit 
minor differences from the version typed into IMDb. 



We also ran experiments relaxing the single-sentence as- 
sumption, which allows for stricter scene control and a larger 
dataset but complicates comparisons involving syntax. The 
non-syntax results were in line with those reported here. 



Movie 


First Quote 


Second Quote 


Jackie Brown 


Half a million dollars will always be missed. 


1 know the type, trust me on this. 


Star Trek: Nemesis 


1 think it's time to ti-y some unsafe velocities. 


No cold feet, or any other parts of our 
anatomy. 


Ordinary People 


A little advice about feelings kiddo; don't ex- 
pect it always to tickle. 


1 mean there's someone besides your 
mother you've got to forgive. 



Table 1: Three example pairs of movie quotes. Each pair satisfies our criteria: the two component quotes are spoken 
close together in the movie by the same character, have the same length, and one is labeled memorable by the IMDb 
while the other is not. (Contractions such as "it's" count as two words.) 



(iii) N does not occur in the IMDb list of memorable 
quotes for the movie (either as a single line or as part 
of a larger block). 

Given such pairs, we formulate a pairwise com- 
parison task: given M and N, determine which is 
the memorable quote. Psychological research on 
subjective evaluation f35l, as well as initial experi- 
ments using ourselves as subjects, indicated that this 
pairwise set-up easier to work with than simply pre- 
senting a single sentence and asking whether it is 
memorable or not; the latter requires agreement on 
an "absolute" criterion for memorability that is very 
hard to impose consistently, whereas the former sim- 
ply requires a judgment that one quote is more mem- 
orable than another. 

Our main dataset, available at 'http://www.es. I 
|cornell.edu/~cristian/memorability.html|j thus con- 
sists of approximately 2200 such (M, A^) pairs, sep- 
arated by a median of 5 same-character lines in the 
script. The reader can get a sense for the nature of 
the data from the three examples in Table [T] 

We now discuss two further aspects to the formu- 
lation of the experiment: a preliminary pilot study 
involving human subjects, and the incorporation of 
search engine counts into the data. 

2.2 Pilot study: Human performance 

As a preliminary consideration, we did a small pilot 
study to see if humans can distinguish memorable 
from non-memorable quotes, assuming our IMDB- 
induced labels as gold standard. Six subjects, all na- 
tive speakers of English and none an author of this 
paper, were presented with 11 or 12 pairs of mem- 
orable vs. non-memorable quotes; again, we con- 
trolled for extra-textual effects by ensuring that in 
each pair the two quotes come from the same movie, 
are by the same character, have the same length, and 

' Also available there: other examples and factoids. 



subject 


number of matches with 




IMDb-induced annotation 


A 


11/11 = 100% 


B 


11/12= 92% 


C 


9/1 1 = 82% 


D 


8/11= 73% 


E 


7/11= 64% 


F 


7/12= 58% 


macro avg 


— 78% 



Table 2: Human pilot study: number of matches to 
IMDb-induced annotation, ordered by decreasing match 
percentage. For the null hypothesis of random guessing, 
these results are statistically significant, p < 2^^ « .016. 

appear as nearly as possible in the same scenejj The 
order of quotes within pairs was randomized. Im- 
portantly, because we wanted to understand whether 
the language of the quotes by itself contains signals 
about memorability, we chose quotes from movies 
that the subjects said they had not seen. (This means 
that each subject saw a different set of quotes.) 
Moreover, the subjects were requested not to consult 
any external sources of information]^ The reader is 
welcome to try a demo version of the task at http:] 
//www.cs.cornell.edu/~cristian/memorability.htmI] 

Table [2] shows that all the subjects performed 
(sometimes much) better than chance, and against 
the null hypothesis that all subjects are guessing ran- 
domly, the results are statistically significant, p < 
2"^ ss .016. These preliminary findings provide ev- 
idence for the validity of our task: despite the appar- 
ent difficulty of the job, even humans who haven't 
seen the movie in question can recover our IMDb- 



In this pilot study, we allowed multi-sentence quotes. 
^We did not use crowd-sourcing because we saw no way to 
ensure that this condition would be obeyed by arbitrary subjects. 
We do note, though, that after our research was completed and 
as of Apr. 26, 2012, ~ 11, 300 people completed the online test: 
average accuracy: 72%, mode number correct: 9/12. 



induced labels with some reliability|j 

2.3 Incorporating search engine counts 

Thus far we have discussed a dataset in which mem- 
orability is determined through an explicit label- 
ing drawn from the IMDb. Given the "produc- 
tion" aspect of memorability discussed in ^ we 
should also expect that memorable quotes will tend 
to appear more extensively on Web pages than non- 
memorable quotes; note that incorporating this in- 
sight makes it possible to use the (implicit) judg- 
ments of a much larger number of people than are 
represented by the IMDb database. It therefore 
makes sense to try using search-engine result counts 
as a second indication of memorability. 

We experimented with several ways of construct- 
ing memorability information from search-engine 
counts, but this proved challenging. Searching for 
a quote as a stand-alone phrase runs into the prob- 
lem that a number of quotes are also sentences that 
people use without the movie in mind, and so high 
counts for such quotes do not testify to the phrase's 
status as a memorable quote from the movie. On 
the other hand, searching for the quote in a Boolean 
conjunction with the movie's title discards most of 
these uses, but also eliminates a large fraction of 
the appearances on the Web that we want to find: 
precisely because memorable quotes tend to have 
widespread cultural usage, people generally don't 
feel the need to include the movie's title when in- 
voking them. Finally, since we are dealing with 
roughly 1000 movies, the result counts vary over an 
enormous range, from recent blockbusters to movies 
with relatively small fan bases. 

In the end, we found that it was more effective to 
use the result counts in conjunction with the IMDb 
labels, so that the counts played the role of an ad- 
ditional filter rather than a free-standing numerical 
value. Thus, for each pair (M, N) produced using 
the IMDb methodology above, we searched for each 
of M and N as quoted expressions in a Boolean con- 
junction with the title of the movie. We then kept 
only those pairs for which M (i) produced more than 
five results in our (quoted, conjoined) search, and (ii) 
produced at least twice as many results as the cor- 



responding search for N. We created a version of 
this filtered dataset using each of Google and Bing, 
and all the main findings were consistent with the 
results on the IMDb-only dataset. Thus, in what fol- 
lows, we will focus on the main IMDb-only dataset, 
discussing the relationship to the dataset filtered by 
search engine counts where relevant (in which case 
we will refer to the -i-Google dataset). 

3 Never send a human to do a machine's job. 

We now discuss experiments that investigate the hy- 
potheses discussed in ^ In particular, we devise 
methods that can assess the distinctiveness and gen- 
erality hypotheses and test whether there exists a no- 
tion of "memorable language" that operates across 
domains. In addition, we evaluate and compare the 
predictive power of these hypotheses. 

3.1 Distinctiveness 

One of the hypotheses we examine is whether the 
use of language in memorable quotes is to some ex- 
tent unusual. In order to quantify the level of dis- 
tinctiveness of a quote, we take a language-model 
approach: we model "common language" using 
the newswire sections of the Brown corpus |2lQ 
and evaluate how distinctive a quote is by evaluat- 
ing its likelihood with respect to this model — the 
lower the likelihood, the more distinctive. In or- 
der to assess different levels of lexical and syntactic 
distinctiveness, we employ a total of six Laplace- 
smootheq^ language models: 1-gram, 2-gram, and 
3-gram word LMs and 1-gram, 2-gram and 3-gram 
part-of-speecqjLMs. 

We find strong evidence that from a lexical per- 
spective, memorable quotes are more distinctive 
than their non-memorable counterparts. As indi- 
cated in Table [3] for each of our lexical "common 
language" models, in about 60% of the quote pairs, 
the memorable quote is more distinctive. 

Interestingly, the reverse is true when it comes to 



The average accuracy being below 100% reinforces that 
context is very important, too. 



'Results were qualitatively similar if we used the fiction por- 
tions. The age of the Brown corpus makes it less likely to con- 
tain modem movie quotes. 

^We employ Laplace (additive) smoothing with a smoothing 
parameter of 0.2. The language models' vocabulary was that of 
the entire training corpus. 

'Throughout we obtain part-of-speech tags by using the 
NLTK maximum entropy tagger with default parameters. 



"common language" 
model 


IMDb-only 


H-Google 


1-gram 

lexical 2-gram 

3 -gram 


61.13%*** 
59.22%*** 
59.81%*** 


59.21%*** 
57.03%*** 

58.32%*** 


1-gram 

syntactic 2-gram 

3 -gram 


43.60%*** 

48.31% 

50.91% 


44.77%*** 

47.84% 

50.92% 



Generality metric 


IMDb-only 


H-Google 


fewer 3"* pers. pronouns 


64.37%*** 


62.93%*** 


more indef. article 


57.21%*** 


58.23%*** 


less past tense 


57.91%*** 


59.74%*** 


more present tense 


54.60%*** 


55.86%*** 



Table 3: Distinctiveness: percentage of quote pairs 
in which the the memorable quote is more distinctive 
than the non-memorable one according to the respec- 
tive "common language" model. Significance accord- 
ing to a two-tailed sign test is indicated using * -notation 
(***="p<.001"). 

syntax: memorable quotes appear to follow the syn- 
tactic patterns of "common language" as closely as 
or more closely than non-memorable quotes. To- 
gether, these results suggest that memorable quotes 
consist of unusual word sequences built on common 
syntactic scaffolding. 

3.2 Generality 

Another of our hypotheses is that memorable quotes 
are easier to use outside the specific context in which 
they were uttered — that is, more "portable" — and 
therefore exhibit fewer terms that refer to those set- 
tings. We use the following syntactic properties as 
proxies for the generality of a quote: 

• Fewer 3'"''-person pronouns, since these com- 
monly refer to a person or object that was intro- 
duced earlier in the discourse. Utterances that 
employ fewer such pronouns are easier to adapt 
to new contexts, and so will be considered more 
general. 

• More indefinite articles like a and an, since 
they are more likely to refer to general concepts 
than definite articles. Quotes with more indefi- 
nite articles will be considered more general. 

• Fewer past tense verbs and more present 
tense verbs, since the former are more likely 
to refer to specific previous events. Therefore 
utterances that employ fewer past tense verbs 
(and more present tense verbs) will be consid- 
ered more general. 

Table |4] gives the results for each of these four 
metrics — in each case, we show the percentage of 



Table 4: Generality: percentage of quote pairs in which 
the memorable quote is more general than the non- 
memorable ones according to the respective metric. Pairs 
where the metric does not distinguish between the quotes 
are not considered. 



quote pairs for which the memorable quote scores 
better on the generality metric. 

Note that because the issue of generality is a com- 
plex one for which there is no straightforward single 
metric, our approach here is based on several prox- 
ies for generality, considered independently; yet, as 
the results show, all of these point in a consistent 
direction. It is an interesting open question to de- 
velop richer ways of assessing whether a quote has 
greater generality, in the sense that people intuitively 
attribute to memorable quotes. 

3.3 "Memorable" language beyond movies 

One of the motivating questions in our analysis 
is whether there are general principles underlying 
"memorable language." The results thus far suggest 
potential families of such principles. A further ques- 
tion in this direction is whether the notion of mem- 
orability can be extended across different domains, 
and for this we collected (and distribute on our web- 
site) 431 phrases that were explicitly designed to 
be memorable: advertising slogans (e.g., "Quality 
never goes out of style."). The focus on slogans is 
also in keeping with one of the initial motivations 
in studying memorability, namely, marketing appli- 
cations — in other words, assessing whether a pro- 
posed slogan has features that are consistent with 
memorable text. 

The fact that it's not clear how to construct a col- 
lection of "non-memorable" counterparts to slogans 
appears to pose a technical challenge. However, we 
can still use a language-modeling approach to as- 
sess whether the textual properties of the slogans are 
closer to the memorable movie quotes (as one would 
conjecture) or to the non-memorable movie quotes. 
Specifically, we train one language model on memo- 
rable quotes and another on non-memorable quotes 



(Non)memorable 
language models 


Slogans 


Newswire 


1-gram 

lexical 2-gram 

3-gram 


56.15%** 
51.51% 

52.44% 


33.77%*** 
25.15%*** 
28.89%*** 


1-gram 

syntactic 2-gram 

3-gram 


73.09%*** 
64.04%*** 
62.88%*** 


68.27%*** 

50.21% 

55.09%*** 



Table 5: Cross-domain concept of "memorable" lan- 
guage: percentage of slogans that have higher likelihood 
under the memorable language model than under the non- 
memorable one (for each of the six language models con- 
sidered). Rightmost column: for reference, the percent- 
age of newswire sentences that have higher likelihood un- 
der the memorable language model than under the non- 
memorable one. 



Generality metric 



slogans 



% Supers, pronouns 2.14% 
% indefinite articles 2.68% 
% past tense 14.60% 



mem. 



n-mem. 



2.16% 3.41% 

2.63% 2.06% 

21.13% 26.69% 



Table 6: Slogans are most general when compared to 
memorable and non-memorable quotes. (%s of 3'^'' pers. 
pronouns and indefinite articles are relative to all tokens, 
%s of past tense are relative to all past and present verbs.) 



and compare how likely each slogan is to be pro- 
duced according to these two models. As shown in 
the middle column of Table [5] we find that slogans 
are better predicted both lexically and syntactically 
by the former model. This result thus offers evi- 
dence for a concept of "memorable language" that 
can be applied beyond a single domain. 

We also note that the higher likelihood of slogans 
under a "memorable language" model is not simply 
occurring for the trivial reason that this model pre- 
dicts all other large bodies of text better. In partic- 
ular, the newswire section of the Brown corpus is 
predicted better at the lexical level by the language 
model trained on non-memorable quotes. 

Finally, Table [6] shows that slogans employ gen- 
eral language, in the sense that for each of our 
generality metrics, we see a slogans/memorable- 
quotes/non-memorable quotes spectrum. 

3.4 Prediction task 

We now show how the principles discussed above 
can provide features for a basic prediction task, cor- 
responding to the task in our human pilot study: 



given a pair of quotes, identify the memorable one. 

Our first formulation of the prediction task uses 
a standard bag-of-words modej^ If there were 
no information in the textual content of a quote 
to determine whether it were memorable, then an 
SVM employing bag-of-words features should per- 
form no better than chance. Instead, though, it ob- 
tains 59.67% (10-fold cross-validation) accuracy, as 
shown in Table [7] We then develop models using 
features based on the measures formulated earlier 
in this section: generality measures (the four listed 
in Table |4]); distinctiveness measures (likelihood ac- 
cording to 1, 2, and 3-gram "common language" 
models at the lexical and part-of-speech level for 
each quote in the pair, their differences, and pair- 
wise comparisons between them); and similarity- 
to-slogans measures (likelihood according to 1, 2, 
and 3-gram slogan-language models at the lexical 
and part-of-speech level for each quote in the pair, 
their differences, and pairwise comparisons between 
them). 

Even a relatively small number of distinctive- 
ness features, on their own, improve significantly 
over the much larger bag-of-words model. When 
we include additional features based on generality 
and language-model features measuring similarity to 
slogans, the performance improves further (last line 
of Table |7]). 

Thus, the main conclusion from these prediction 
tasks is that abstracting notions such as distinctive- 
ness and generality can produce relatively stream- 
lined models that outperform much heavier-weight 
bag-of-words models, and can suggest steps toward 
approaching the performance of human judges who 

— very much unlike our system — have the full cul- 
tural context in which movies occur at their disposal. 

3.5 Other characteristics 

We also made some auxiliary observations that may 
be of interest. Specifically, we find differences in let- 
ter and sound distribution (e.g., memorable quotes 

— after curse-word removal — use significantly 
more "front sounds" (labials or front vowels such 
as represented by the letter /) and significantly fewer 
"back sounds" such as the one represented by u)Y\ 



10 



We discarded terms appearing fewer than 10 times. 
"These findings may relate to marketing research on sound 
symbolism f7l[T9ll40l. 



Feature set 


# feats 


Accuracy 


bag of words 


962 


59.67% 


distinctiveness 

generality 

slogan sim. 

all three types together 


24 

4 

24 

52 


62.05%* 

56.70% 

58.30% 

64.27%** 



Table 7: Prediction: SVM 10-fold cross validation results 
using the respective feature sets. Random baseline accu- 
racy is 50%. Accuracies statistically significantly greater 
than bag-of-words according to a two-tailed t-test are in- 
dicated with *(p<.05) and **(p<.01). 

word complexity (e.g., memorable quotes use words 
with significantly more syllables) and phrase com- 
plexity (e.g., memorable quotes use fewer coordi- 
nating conjunctions). The latter two are in line with 
our distinctiveness hypothesis. 

4 A long time ago, in a galaxy far, far away 

How an item's linguistic form affects the reaction it 
generates has been studied in several contexts, in- 
cluding evaluations of product reviews (9^\, political 
speeches Iil2il . on-line posts [13], scientific papers 
|[T4l . and retweeting of Twitter posts [36]. We use 
a different set of features, abstracting the notions of 
distinctiveness and generality, in order to focus on 
these higher- level aspects of phrasing rather than on 
particular lower-level features. 

Related to our interest in distinctiveness, work in 
advertising research has studied the effect of syntac- 
tic complexity on recognition and recall of slogans 
in m 24J. There may also be connections to Von 
Restorff's isolation effect Hunt |17], which asserts 
that when all but one item in a list are similar in some 
way, memory for the different item is enhanced. 

Related to our interest in generality, Knapp et al. 
ll20l surveyed subjects regarding memorable mes- 
sages or pieces of advice they had received, finding 
that the ability to be applied to multiple concrete sit- 
uations was an important factor. 

Memorability, although distinct from "memoriz- 
ability", relates to short- and long-term recall. Thorn 
and Page |'34'| survey sub-lexical, lexical, and se- 
mantic attributes affecting short-term memorability 
of lexical items. Studies of verbatim recall have also 
considered the task of distinguishing an exact quote 
from close paraphrases [3|. Investigations of long- 
term recall have included studies of culturally signif- 



icant passages of text ||29ll and findings regarding the 
effect of rhetorical devices of alliterative ||4j|, "rhyth- 
mic, poetic, and thematic constraints" |[T8ll26l . 

Finally, there are complex connections between 
humor and memory ll32ll . which may lead to interac- 
tions with computational humor recognition ||25l . 

5 I think this is the beginning of a 
beautiful friendship. 

Motivated by the broad question of what kinds of in- 
formation achieve widespread public awareness, we 
studied the the effect of phrasing on a quote's mem- 
orability. A challenge is that quotes differ not only 
in how they are worded, but also in who said them 
and under what circumstances; to deal with this dif- 
ficulty, we constructed a controlled corpus of movie 
quotes in which lines deemed memorable are paired 
with non-memorable hnes spoken by the same char- 
acter at approximately the same point in the same 
movie. After controlling for context and situation, 
memorable quotes were still found to exhibit, on av- 
erage (there will always be individual exceptions), 
significant differences from non-memorable quotes 
in several important respects, including measures 
capturing distinctiveness and generality. Our ex- 
periments with slogans show how the principles we 
identify can extend to a different domain. 

Future work may lead to applications in market- 
ing, advertising and education [4}. Moreover, the 
subtle nature of memorability, and its connection to 
research in psychology, suggests a range of further 
research directions. We believe that the framework 
developed here can serve as the basis for further 
computational studies of the process by which infor- 
mation takes hold in the public consciousness, and 
the role that language effects play in this process. 

My mother thanks you. My father thanks you. 
My sister thanks you. And I thank you: Re- 
becca Hwa, Evie Kleinberg, Diana Minculescu, Alex 
Niculescu-Mizil, Jennifer Smith, Benjamin Zimmer, and 
the anonymous reviewers for helpful discussions and 
comments; our annotators Steven An, Lars Backstrom, 
Eric Baumer, Jeff Chadwick, Evie Kleinberg, and Myle 
Ott; and the makers of Cepacol, Robitussin, and Sudafed, 
whose products got us through the submission deadline. 
This paper is based upon work supported in part by NSF 
grants IIS-0910664, IIS-1016099, Google, and Yahoo! 
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